Social Networking System Targeted Message Synchronization

ABSTRACT

Targeted messages are sent to users of social networking system (SNS) based on the detection of airings of advertisements in time-based media. This approach allows advertisers to leverage their investment in, for example, television advertising by sending advertisements to SNS users who have likely seen the advertisements within shows that they have expressed interest in the context of a SNS. Time-based media streams are analyzed to detect the airings of advertisements within those streams. In one embodiment, SNS content items are received regarding individual users. Based on references in those content items between users and their interests, targeted messages may be sent to users. In another embodiment, targeted messages are sent to SNS users based on the airing of advertisements and targeting criteria provided by advertisers.

FIELD OF THE INVENTION

The present invention relates generally to correlating advertisementswith video content and using those correlations to send targetedadvertisements to social networking system users.

BACKGROUND OF THE INVENTION

Online social media services, such as social networking sites, searchengines, news aggregators, blogs, and the like provide a richenvironment for users to comment on events of interest and communicatewith other users. Social media content items authored by users of socialnetworking systems often include references to events that appear intime based media such as television shows, news reports, sportingevents, movies, concert performances, and the like. However, althoughthe content items can sometimes refer to the time-based media, thesocial media content items themselves typically are isolated from theevents and time-based media that those content items refer to.

SUMMARY OF THE INVENTION

Targeted messages are sent to users of social networking system (SNS)based on the detection of airings of advertisements in time-based media.This approach allows advertisers to leverage their investment in, forexample, television advertising by sending advertisements to SNS userswho have likely seen the advertisements within shows that they haveexpressed interest in the context of a SNS.

In one embodiment, one or more television streams are monitored todetect when and where a specific advertisement for a particularadvertiser is shown. Metadata associated with the television shows andads, for example the show's title, character names, actor names, plotaspects, or the like, is stored. The TV streams are segmented, and usingthe metadata individual video events are determined. The video eventsinclude the airing locations and times of individual TV shows andadvertisements, so that it can be determined which advertisements airedduring which TV shows.

In one embodiment, social media content items are received from a SNS.The social media content items contain content and one or morereferences to other content items associated with users of the SNS orwith specific TV shows or advertisements. The references between contentitems identify the TV shows and advertisements that individual SNS usersare connected to in the context of the SNS. The references in socialmedia content items and the airings of advertisements within TV showsare used to create mappings between SNS users and the advertisementsthey are likely to have seen. Mappings may also be used to createpopulations of users who are likely to have seen a given advertisementor TV show. The users making up a population may also be filtered basedon one or more of demographic, content, or time criteria. Responsive toa specific advertisement being detected as being aired during a specificTV show, or responsive to a request for a message from the SNS, amessage is sent to one or more SNS users based on the mappings.

In another embodiment, the detection of airings of advertisements withintelevision shows are used to send targeted messages to SNS users. Themessages may be sent along with targeting criteria specifying which SNSusers are designated to receive the sent messages. Messages may be sentto the SNSs at the initiative of the targeted messaging provider, or atthe request of the SNS. Although described with respect to televisionshows particularly, the systems and processes described herein may beused in conjunction with any form of time-based media.

The features and advantages described in this summary and the followingdetailed description are not all-inclusive. Many additional features andadvantages will be apparent to one of ordinary skill in the art in viewof the drawings, specification, and claims hereof.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates the computing environment of one embodiment of asystem for associating sending targeted messages to social networkingsystem users.

FIG. 2 is a block diagram of one embodiment of a targeted messagingprovider.

FIG. 3 is a conceptual diagram illustrating the video/metadata alignmentprocess at a high level according to one embodiment.

FIG. 3A is a flow diagram illustrating one embodiment of a method fordetermining the airings of time-based media events.

FIG. 4 is a flow diagram illustrating one embodiment of a video eventsegmentation process.

FIG. 5 is a flow diagram illustrating one embodiment of a video metadataalignment process.

FIG. 6 is a flow diagram illustrating one embodiment of socialnetworking system targeted message synchronization.

FIG. 7 is a flow diagram illustrating one embodiment of a method forsocial networking system (SNS) targeted message synchronization.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION

FIG. 1 illustrates the computing environment 100 of one embodiment of asystem 130 for associating sending targeted messages to socialnetworking system users based on associations between advertisements andother time-based media.

The environment 100 includes social networking systems (SNSs) 110,time-based media sources 120, the targeted messaging provider 130, anetwork 140, client devices 150, and advertisement sources 160.

The SNSs 110 include social networks, blogs, news media, forums, usergroups, etc. These systems generally provide a plurality of users withthe ability to communicate and interact with other users of the system.Users can typically contribute various social media content items (e.g.,posts, videos, photos, links, status updates, blog entries, tweets,profiles, and the like), which may refer to media events (e.g., TVshows, advertisements) or other social media content items (e.g., pagesassociated with TV shows or advertisements), and can engage indiscussions, games, online events, and other participatory services.

The time-based media sources 120 include broadcasters, direct contentproviders, advertisers, and any other third-party providers oftime-based media content. These sources 120 typically publish contentsuch as TV shows, videos, movies, serials, audio recordings, and thelike.

The network 140 may comprise any combination of local area and/or widearea networks, the Internet, or one or more intranets, using both wiredand wireless communication systems.

The client devices 150 comprise computing devices that can receive inputfrom a user and can transmit and receive data via the network 140. Forexample, client devices 150 may be a desktop computer, a laptopcomputer, a smart phone, a personal digital assistant (PDAs), or anyother device including computing functionality and data communicationcapabilities. A client device 150 is configured to communicate with thesocial networking systems 110 via the network 140.

The advertisement sources 160 include companies, advertising agencies,or any other third-party organizations that create messages (e.g.,advertisements, creatives) to be sent to SNS users through the SNS 110.The messages may be published in the SNSs 110 alongside other content,for example in the web page of a browser viewed by a SNS user on aclient device 150, however the messages may also be displayed alongsidetime-based media sources 120 (e.g., TV shows, audio recordings). Themessages may be provided to the targeted messaging provider 130 to besent to the SNS 110 at the discretion of the provider 130. The messagesmay also be provided directly to SNSs 110, and may be sent to SNS usersin response to other messages from the targeted messaging provider 130,

The targeted messaging provider 130 provides targeted advertisements tousers of social networking systems based information received from thetime-based media sources 120 and the SNSs 110, and is further describedin conjunction with FIGS. 2-7.

FIG. 2 is a block diagram of one embodiment of a targeted messagingprovider 130. The targeted messaging provider 130 shown in FIG. 2 is acomputer system that includes a web server 200 and associated API 202, aclosed captioning extractor 305, a video event segmentation engine 310,a feature extraction engine 315, a video metadata alignment engine 320,a TV show/ad overlap engine 615, an audience population engine 625, amessage selection engine 635, a messaging interface 645, a closedcaptioning store 267, a multimedia store 270, an event metadata store273, a video event store 280, an annotated event store 290, a receivedsocial media information store 260, a TV programming guide 605, a TVshow/ad overlap engine 620, a population store 630, and a messagelibrary 640.

This system may be implemented using a single computer, or a network ofcomputers, including cloud-based computer implementations. The computersare preferably server class computers including one or morehigh-performance CPUs, 1G or more of main memory, as well as 500 GB to 2Tb of computer readable, persistent storage, and running an operatingsystem such as LINUX or variants thereof. The operations of the system130 as described can be controlled through either hardware or throughcomputer programs installed in computer storage and executed by theprocessors of such servers to perform the functions described herein.The system 130 includes other hardware elements necessary for theoperations described here, including network interfaces and protocols,security systems, input devices for data entry, and output devices fordisplay, printing, or other presentations of data; these and otherconventional components are not shown so as to not obscure the relevantdetails.

As noted above, system 130 comprises a number of “engines,” which refersto computational logic for providing the specified functionality. Anengine can be implemented in hardware, firmware, and/or software. Anengine may sometimes be equivalently referred to as a “module” or a“server.” It will be understood that the named components represent oneembodiment of the present invention, and other embodiments may includeother components. In addition, other embodiments may lack the componentsdescribed herein and/or distribute the described functionality among thecomponents in a different manner. Additionally, the functionalitiesattributed to more than one component can be incorporated into a singlecomponent. Where the engines described herein are implemented assoftware, the engine can be implemented as a standalone program, but canalso be implemented through other means, for example as part of a largerprogram, as a plurality of separate programs, or as one or morestatically or dynamically linked libraries. In any of these softwareimplementations, the engines are stored on the computer readablepersistent storage devices of the system 130, loaded into memory, andexecuted by the one or more processors of the system's computers. Theoperations of the system 130 and its various components will be furtherdescribed below with respect to FIG. 2 and the remaining figures. Aswill become apparent, the various data processing operations describedherein are sufficiently complex and time consuming as to require theoperation of a computer system such as the system 130.

The web server 200 links the system 130 to the client devices 150, thetime-based media sources 120, and the social networking systems 110 vianetwork 140, and is one means for doing so. The web server 200 servesweb pages, as well as other web related content, such as Java, Flash,XML, and so forth. The web server 200 may include a mail server or othermessaging functionality for receiving and routing messages between thesystem 130 and client devices 150.

The API 202, in conjunction with web server 200, allows one or moreexternal entities to access information from the system 130. The webserver 200 may also allow external entities to send information to thesystem 130 calling the API 202. For example, an external entity sends anAPI request to the system 130 via the network 140 and the web server 200receives the API request. The web server 200 processes the request bycalling an API 202 associated with the API request to generate anappropriate response, which the web server 200 communicates to theexternal entity via the network 140. The API may be used by a SNS 110 tocommunicate information and requests to the system 130.

The closed captioning extractor 305 extracts closed captioning data fromthe time-based media. Closed captioning data typically can be extractedfrom broadcast video or other sources encoded with closed captions usingopen source software such as CCExtractor available via SourceForge.net.For time-based media not encoded with closed captioning data, imperfectmethods such as automatic speech recognition can be used to capture andconvert the audio data into a text stream comparable to closedcaptioning text. This can be done, for example, using open sourcesoftware such as Sphinx 3 available via SourceForge.net. Once the closedcaptioning is ingested, it is preferably aligned to speech in a video.Various alignment methods are known in the art. One such method isdescribed in Hauptmann, A. and Witbrock, M., Story Segmentation andDetection of Commercials in Broadcast News Video, ADL-98 Advances inDigital Libraries Conference, Santa Barbara, Calif. (April 1998), whichuses dynamic programming to align words in the closed captioning streamto the output of a speech recognizer run over the audio track of thevideo. The closed captioning information is stored in the closedcaptioning store 267.

The multimedia store 270 stores various forms of time-based media.Time-based media includes any data that changes meaningfully withrespect to time. Examples include, and are not limited to, videos,(e.g., TV shows or portions thereof, movies or portions thereof) audiorecordings, MIDI sequences, animations, and combinations thereof.Time-based media can be obtained from a variety of sources, such aslocal or network stores, as well as directly from capture devices suchas cameras, microphones, and live broadcasts. It is anticipated thatother types of time-based media within the scope of the invention willbe developed in the future (e.g., 3D media, holographic presentations,immersive media, and so forth).

The video event segmentation engine 310 segments time-based media intosemantically meaningful segments corresponding to discrete portions or“events” (e.g., advertisement events, television events, etc.) and isone means for doing so. Although described with respect to video, thevideo event segmentation 310 may also operate on audio media, such asfor radio broadcasts. The video event segmentation process includesthree main components according to one embodiment: shot boundarydetection, event detection, and boundary determination. These componentsfor event segmentation may vary by domain (e.g., video, audio). Theoutput of video event segmentation is a set of segmented video eventsthat is stored in the video event store 280.

The feature extraction engine 315 converts segmented time-based mediaevents retrieved from the video event store 280 into feature vectorrepresentations for aligning the events with metadata, and is one meansfor doing so. The features may include image and audio properties andmay vary by domain. Feature types may include, but are not limited to,scale-variant feature transform (SIFT), speeded up robust features(SURF), local energy based shape histogram (LESH), color histogram, andgradient location orientation histogram (GLOH).

The video metadata alignment engine 320 aligns video event segments withsemantically meaningful information regarding the advertisement eventthat the event is about, and is one means for doing so. As with thevideo event segmentation engine 310, the video metadata alignment mayalso operate on audio events 320. The video metadata alignment engine320 uses metadata instances from the event metadata store 273. Ametadata instance is the metadata for a single event, i.e., a singlepiece of metadata. The video metadata alignment engine 320 may also beused to annotate the segments with the metadata, or alternatively aseparate annotation engine (not shown) may be used. Metadata instancesmay include automatic annotations of low level content features, e.g.,image or audio features or content features, hand annotations with textdescriptions, or both. The metadata may be represented as textdescriptions of time-based media events and feature vectorrepresentations extracted from examples of events. The annotations arestored in the annotated event store 290.

The social media store 260 stores social media content items receivedfrom SNSs 110. In general, SNSs 110 allow their users to publish contentitems to other members of their network, which may be open and viewableby the public through open application program interfaces. Social mediacontent items include long form and short form items such as posts,videos, photos, links, status updates, blog entries, tweets, and thelike. Other examples of social media content items include audio ofcommentators on, or participants of, another event or topic (e.g.,announcers on TV or radio) and text transcriptions thereof (generatedmanually or automatically), event-related information (e.g., recipes,instructions, scripts, etc.), statistical data (e.g., sports statisticsor financial data streams), news articles, and media usage statistics(e.g., user behavior such as viewing, rewind, pausing, etc.).

The TV show/ad overlap engine 615, audience population engine 625,message selection engine 635, messaging interface 645, TV programmingguide 605, TV show/ad overlap store 620, population store 630, andmessage library 640 are described below with respect to FIGS. 6 and 7.

Determining Airings of Time-Based Media Events

FIG. 3 is a conceptual diagram illustrating the video metadata alignment320 process at a high level according to one embodiment. Although thisprocess is described with respect to video events, it can also be usedin conjunction with audio events. Beginning with metadata instances 307and events in time-based media 301 as input, annotated events 309 areformed. As shown, time-based media (TBM) 301 includes multiple segments(seg. 1-M) 303, which contain events in the time-based media, asdescribed herein. The video/metadata alignment 320 process aligns one ormore metadata instances (1-N) 307 with the events to form annotatedevents 309, as further described in conjunction with FIG. 5. Note thatin process 320 the various alignments are one-to-one, many-to-one,and/or many-to-many. Once so mapped, the relationships betweenadvertisement media events and non-advertisement media events can bedetermined to send targeted messaging, as further explained below.

FIG. 3A is a flow diagram illustrating one embodiment of a method fordetermining the airings of time-based media events.

As a preliminary step in the method, multiple streams of data areingested 300 at the system 130 for processing. Data may be received atthe system 130 directly from content providers, or from socialnetworking systems 110 or time-based media sources 120, e.g., frombroadcast television feeds, radio feeds, internet streams, directly fromcontent producers, and/or from other third parties. In one embodiment,web server 200 is one means for ingesting 300 the data. The types ofdata may include, but are not limited to, time-based media, electronicprogramming guide data 605 metadata, closed captioning data, statistics,social media posts, mainstream news media, and usage statistics, such asdescribed above.

The ingested data is stored in data stores specific to one or more datatypes that serve as the input data sources for the primary processes ofthe method of FIG. 3A (each shown in bold). For example, time-basedmedia data is stored in the multimedia store 270. The time-based mediain the multimedia store 270 may undergo additional processing beforebeing used within the methods shown in FIGS. 3-6. For example, closedcaptioning data can be extracted from 305 the time-based media, e.g., byclosed captioning extractor 305. In addition, event metadata associatedwith multimedia is stored in the event metadata store 273, and socialmedia information in the received social media information store 260.

As a result of the ingestion referenced above, the multimedia store 270includes various forms of time-based media. The time-based media may beof various types, as described in conjunction with FIG. 2.

The process for detecting the airings of various time based media eventsmay be described as including two major processes: video eventsegmentation 310 and video metadata alignment 320. Each of theseprocesses 310-320 are described below. The process of FIG. 3A may bereferred to as an event airing detection process 610.

Video Event Segmentation

The first process is video event segmentation 310, in which thetime-based media is segmented into semantically meaningful segmentscorresponding to discrete events depicted in video at semanticallymeaningful boundaries. The input to the video event segmentation 310process is a raw video (and/or audio) stream that is retrieved from themultimedia store 270 according to one embodiment, and may be performed,e.g., by the event segmentation engine 220.

The video event segmentation 310 process includes three main componentsaccording to one embodiment: shot boundary detection, event detection,and boundary determination. These components may vary by domain. Forexample, for sporting events an additional component may correspond toscene classification (e.g., field or stadium identification).

The output of video event segmentation 310 is a set of segmented videoevents that are stored in the video event store 280. Video eventsegmentation 310 is described in further detail in conjunction with FIG.4.

Video Metadata Alignment

The next process is video metadata alignment 320, in which the segmentsfrom video event segmentation 310 are annotated with semanticallymeaningful information regarding the event that the segment is relevantto, or depicts. Input to video metadata alignment 320 is a video eventretrieved from the video event store 280 and metadata from the eventmetadata store 273. Such metadata can include, but is not limited to:the type of event occurring, the brand/product for which anadvertisement event is advertising, the agents actors/charactersinvolved in the event, the scene/location of the event, the time lengthof the event, the results/causes of the event, etc. For example,metadata for an advertisement event may include information such as“Brand: Walmart, Scene: father dresses up as clown, Mood: comic.” Asillustrated in these examples, the metadata can be structured as tuplesof <name, value> pairs.

The metadata includes text and lower level image and audio properties.Metadata may be generated using human annotation (e.g., via humanannotators watching events or samples thereof) and may be supplementedwith automatic annotations for use in the alignment process (e.g.,describing lower level image and audio properties of the event such asnumber and length of each shot, average color histograms of each shot,power levels of the associated audio, etc.) The annotation is stored inthe annotated event store 290.

Video metadata alignment 320 includes two steps according to oneembodiment: event feature extraction and video metadata alignment. Videometadata alignment 320 is described in further detail in conjunctionwith FIG. 5.

According to another embodiment, data ingestion 300, video eventsegmentation 310, and video metadata alignment 320 (or, collectively,event airing detection 610) could be performed by a separate entity,such as a content provider or owner, e.g., which does not want torelease the video content to others. In this embodiment, the targetedmessaging provider 130 would provide software, including the softwaremodules and engines described herein, to the separate entity to allowthem to perform these processes on the raw time-based media. Theseparate entity in return could provide the system 130 with theextracted features, video events, and their respective metadata for useby the system 130. These data exchanges could take place via API 202exposed to the separate entity via web server 200.

Video Event Segmentation

FIG. 4 is a flow diagram illustrating one embodiment of a video eventsegmentation process 310. As described in FIG. 3A, video eventsegmentation 310 segments time-based media into semantically meaningfulsegments corresponding to discrete portions or “events,” e.g., via eventsegmentation engine 220, which is one means for performing thisfunction.

Input to the video event segmentation process 310 is a video stream 405from the multimedia store 270. Video event segmentation 310 includes 3phases: shot boundary detection 410, event detection 420, and eventboundary determination 430, each of which is described in greater detailbelow. The output of video event segmentation 310 is a segmented videoevent 435, which is stored in the video event store 280.

Shot Boundary Detection

The first step in segmenting is shot boundary detection 410 for discretesegments (or “shots”) within a video. Shot boundaries are points ofnon-continuity in the video, e.g., associated with a change in a cameraangle or scene. Shot boundaries may be determined by comparing colorhistograms of adjacent video frames and applying a threshold to thatdifference. Shot boundaries may be determined to exist wherever thedifference in the color histograms of adjacent frames exceeds thisthreshold. Many techniques are known in the art for shot boundarydetection. One exemplary algorithm is described in Tardini et al., ShotDetection and Motion Analysis for Automatic MPEG-7 Annotation of SportsVideos, 13th International Conference on Image Analysis and Processing(November 2005). Other techniques for shot boundary detection 410 may beused as well, such as using motion features. Another known technique isdescribed in A. Jacobs, et al., Automatic shot boundary detectioncombining color, edge, and motion features of adjacent frames, Centerfor Computing Technologies, Bremen, Germany (2004).

Event Detection

Event detection 420 identifies the presence of an event in a stream of(one or more) segments using various features corresponding, forexample, to the image, audio, and/or camera motion for a given segment.A classifier using such features may be optimized by hand or trainedusing machine learning techniques such as those implemented in the WEKAmachine learning package described in Witten, I. and Frank, E., DataMining: Practical machine learning tools and techniques (2nd Edition),Morgan Kaufmann, San Francisco, Calif. (June 2005). The event detectionprocess 420 details may vary by domain.

Image features are features generated from individual frames within avideo. They include low level and higher level features based on thosepixel values. Image features include, but are not limited to, colordistributions, texture measurements, entropy, motion, detection oflines, detection of faces, presence of all black frames, graphicsdetection, aspect ratio, and shot boundaries.

Speech and audio features describe information extracted from the audioand closed captioning streams. Audio features are based on the presenceof music, cheering, excited speech, silence, detection of volume change,presence/absence of closed captioning, etc. According to one embodiment,these features are detected using boosted decision trees. Classificationoperates on a sequence of overlapping frames (e.g., 30 ms overlap)extracted from the audio stream. For each frame, a feature vector iscomputed using Mel-frequency cepstral coefficients (MFCCs), as well asenergy, the number of zero crossings, spectral entropy, and relativepower between different frequency bands. The classifier is applied toeach frame, producing a sequence of class labels. These labels are thensmoothed using a dynamic programming cost minimization algorithm,similar to those used in hidden Markov models.

In addition to audio features, features may be extracted from the wordsor phrases spoken by narrators and/or announcers. From a domain specificontology (not shown), a predetermined list of words and phrases isselected and the speech stream is monitored for the utterance of suchterms. A feature vector representation is created in which the value ofeach element represents the number of times a specific word from thelist was uttered. The presence of such terms in the feature vectorcorrelates with the occurrence of an event associated with thepredetermined list of words. For example, the uttering of the phrase“Travelocity” is correlated with the occurrence of an advertisement forTravelocity.

Unlike image and audio features, camera motion features represent moreprecise information about the actions occurring in a video. The cameraacts as a stand in for a viewer's focus. As actions occur in a video,the camera moves to follow it; this camera motion thus mirrors theactions themselves, providing informative features for eventidentification. Like shot boundary detection, there are various methodsfor detecting the motion of the camera in a video (i.e., the amount itpans left to right, tilts up and down, and zooms in and out). Oneexemplary system is described in Bouthemy, P., et al., A unifiedapproach to shot change detection and camera motion characterization,IEEE Trans. on Circuits and Systems for Video Technology, 9(7) (October1999); this system computes the camera motion using the parameters of atwo-dimensional affine model to fit every pair of sequential frames in avideo. According to one embodiment, a 15-state first-order hidden Markovmodel is used, implemented with the Graphical Modeling Toolkit, and thenthe output of the Bouthemy is output into a stream of clusteredcharacteristic camera motions (e.g., state 12 clusters together motionsof zooming in fast while panning slightly left).

Event Boundary Determination

Once a segment of video is determined to contain the occurrence of anevent, the beginning and ending boundaries of that event must bedetermined 430. In some cases, the shot boundaries determined in 410 areestimates of the beginning and end of an event. The estimates can beimproved as well by exploiting additional features of the video andaudio streams to further refine the boundaries of video segments. Eventboundary determination 430 may be performed using a classifier that maybe optimized by hand or using supervised learning techniques. Theclassifier may make decisions based on a set of rules applied to afeature vector representation of the data. The features used torepresent video overlap with those used in the previous processes.Events have beginning and end points (or offsets), and those boundariesmay be determined based on the presence/absence of black frames, shotboundaries, aspect ratio changes, etc., and have a confidence measureassociated with the segmentation. The result of event boundarydetermination 430 (concluding video event segmentation 410) is a (setof) segmented video event 435 that is stored in the video event store280.

Video Metadata Alignment

FIG. 5 is a flow diagram illustrating one embodiment of a video metadataalignment 320 process. As described in FIG. 3A, the video metadataalignment 320 process produces annotations of the segments from videoevent segmentation 310, which annotations include semanticallymeaningful information regarding the event or topic that the segment isabout. Video metadata alignment 320 includes two steps: featureextraction 315 and video metadata alignment 520.

Video Feature Extraction

For any given video event that is to be aligned with metadata, the firststep is to convert the video event into a feature vector representationvia feature extraction 315. The feature extraction engine 315 is onemeans for performing this function. Input to the process is a segmentedvideo event 435 retrieved from the video event store 280. Output fromthe video feature extraction 315 is a video event feature representation510. The features may be identical to (or a subset of) the image/audioproperties discussed above for video events and stored in the eventmetadata store 273, and may vary by domain.

Video Metadata Alignment

Video metadata alignment 520 takes as input the video feature vectorrepresentation 510 of a video event and a metadata instance 505, definedabove as metadata corresponding to a single event. The video metadataalignment engine 320 is one means for performing this function. Itcycles through each metadata instance 505 in the event metadata store273 and uses an alignment function to estimate the likelihood that aparticular event may be described by a particular metadata instance foran event. As described above, metadata instances include automaticannotations of low level content features (e.g., image or audiofeatures) and hand annotations of text descriptions. The alignmentfunction may be a simple cosign similarity function that compares thefeature representation 510 of the event to the low level propertiesdescribed in the metadata instance 505.

When all metadata instances 505 in the event metadata store 273corresponding to the event have been examined, if the most likelyalignment 525 (i.e., alignment with the highest probability or score)passes a threshold, the video event associated with the featurerepresentation 510 is annotated with the metadata instance 505 and theresulting annotated event 530 is stored in an annotated event store 290along with a score describing the confidence of the annotation. If noevent passes the threshold, the event is marked as not annotated. Inorder to set this threshold, a set of results from the process is handannotated into two categories: correct and incorrect results.Cross-validation may then be used to find the threshold that maximizesthe precision/recall of the system over the manually annotated resultset.

Example: Advertising

As described in conjunction with FIG. 3A, multiples streams of data areingested as a preliminary step in the method.

Video Event Segmentation

For the advertising domain, during the video event segmentation 310process, the time-based media is segmented into semantically meaningfulsegments corresponding to discrete “events” which are identified withadvertisements (i.e. commercials).

Event detection 420 in the advertising domain may operate by identifyingone or more shots that may be part of an advertisement. Advertisementscan be detected using image features such as the presence of all blackframes, graphics detection (e.g. presence of a channel logo in theframe), aspect ratio, shot boundaries, etc. Speech/audio features may beused including detection of volume change, and the presence/absence ofclosed captioning.

Event boundary detection 430 operates on an advertisement block andidentifies the beginning and ending boundaries of individual ads withinthe block. Event boundary determination may be performed using aclassifier based on features such as the presence/absence of blackframes, shot boundaries, aspect ratio changes, typical/expected lengthof advertisements. Classifiers may be optimized by hand or using machinelearning techniques.

Video Metadata Alignment

As with event segmentation 310, the video metadata alignment 320 processis domain dependent. In the advertisement domain, metadata for anadvertisement may include information such as “Brand: Walmart, Scene:father dresses up as clown, Mood: comic.” This metadata is generated byhuman annotators who watch sample ad events and log metadata for ads,including, the key products/brands involved in the ad, the mood of thead, the story/creative aspects of the ad, the actors/celebrities in thead, etc.

Metadata for advertisements may also include low level image and audioproperties of the ad (e.g. number and length of shots, average colorhistograms of each shot, power levels of the audio, etc.).

For each event (i.e., advertisement) that is to be aligned withmetadata, the advertisement is converted into a feature vectorrepresentation via feature extraction 315. Video metadata alignment 520then takes as input the feature vector representation 510 of anadvertisement and a metadata instance 505. It cycles through eachmetadata instance 505 in the event metadata store 273 and estimates thelikelihood that the particular advertisement may be described by aparticular metadata instance using, for example, a simple cosignsimilarity function that compares the low level feature representationof the ad event to the low level properties in the metadata.

The particular start and end times, channel and location in which thespecific advertisement appeared is included with the metadata that isstored in the Annotated Event Store 290.

Social Networking System Targeted Message Synchronization

As advertisers use the audience demographics of TV shows to select TVshows in which to show advertisements, SNS targeted messagesynchronization allows advertisers to leverage their investments inadvertisements with messaging to SNS users who are most likely to be ina particular demographic of interest to the advertiser, and/or who aremost likely to have already viewed their advertisements.

FIG. 6 is a flow diagram illustrating one embodiment of a method forsocial networking system (SNS) targeted message synchronization. In thisembodiment, SNS targeted message synchronization sends messages to usersof SNSs who reference social media content (or are referenced in orassociated with social media content items) that also reference contentitems related to TV shows or advertisements. The SNS targeted messagesynchronization process leverages the assumption that a SNS user whorefers to a TV show or advertisement in a social media content item islikely to have been exposed to the advertisements that are aired duringthat TV show or near in time to that advertisement. SNS targetedmessaging synchronization thus sends messages to users based on theadvertisements they are likely to have already been exposed to.

In one embodiment of a SNS targeted message synchronization process, thedata ingest process 300 accesses data from a number of different datafeeds including video streams and TV electronic programming guide data(“EPG data”). Data ingestion 300 receives the TV streams from networkbroadcast, cable, internet, satellite providers, video hosting services,or other sources of video content. The EPG data is stored in the TVprogramming guide 605 store as a set of mappings between metadata (e.g.TV show names, casts, characters, genres, episode descriptions, etc.)and specific airing information (e.g. time, time zone, channel, network,region, etc.). EPG data can be obtained from broadcast and cablenetworks, multiple system operators, or third party services.

The TV streams are processed using event airing detection 610 in orderto identify airings of specific advertisements and TV shows in the TVstreams. The output of event airing detection 610 is stored in theannotated event store 290 as a set of mappings between video events(e.g., advertisements) and metadata annotations (e.g., showing time,channel, brand, keywords, etc.) associated with those video events.Event airing detection 610 is described generally above with respect toFIG. 3A. Event airing detection 610 incorporates video eventsegmentation 310, which is described in FIG. 4, as well as videometadata alignment/annotation 320, which is described in FIG. 5. Aspecific example of event airing detection 610 is described above underthe heading “Example: Advertising”.

The TV show/ad overlap engine 615 accesses the annotated video events290 and the EPG data 605 to create mappings between the detected airingsof advertisements and the TV shows in which those airings occurred, thatis, to determine which advertisements aired during which TV shows. Thematching may include, for example, comparing the temporal extent of theairing times of the TV shows and advertisements. If an advertisementairs between the total temporal extent of the TV show, the airingadvertisement is determined to match (or overlap) the airing of the TVshow. When an airing of an advertisement occurs on the same channel, inthe same TV market, and within the same airing time window as a TV show,a mapping indicative of this occurrence is stored in the TV show/adoverlap store 620 by the TV show/ad overlap engine 615. For example, amapping may be created between an ad for laundry detergent airing at7:15 pm PST on FOX™ on Comcast™ cable and an episode of the TV show Gleefrom 7:00 pm to 8:00 pm PST, also on FOX™ on Comcast™ cable.

In addition to ingesting video streams and related metadata, the dataingest process 300 also accesses social media information from the SNSsthat the system 130 is configured to provide with targeted messaging. Inone embodiment, the social media information is accessed by the system130 using a SNS API 112. The accessed social media information includesa number of social media content items, that are stored in the socialmedia store 260.

Social media content items may include any object or data storage unitfrom the SNS. Social media content items generally contain contentcreated or added by the user who authors the item, along with one ormore references to other social media content items. Generally, eachitem will include at least a reference to the content item signifyingthe authoring user (e.g., the authoring user's profile in the SNS),however other users (e.g., friends of the user) or objects (e.g., thecontent item associated with a TV show) may also be referenced in acontent item. Thus, a given user may be said to be “referenced in” asocial content item by either being its author, or by being identifiedin, linked to, or a recipient of, a social content item.

One example of a general type of social media content item includescontent items related to specific real world entities. For example,social media content items may be created for individual users (e.g.,user profiles), or for organizations, events, TV shows, advertisingcampaigns, and the like. Each of these content items may include its owncontent (e.g., the name of the user and the user's demographicinformation) and references to other content items (e.g., connections tothe user's interests and to the user's friends).

Another example of a general type of social media content item includeposts by users regarding other social media content items already inexistence in the SNS. Examples of these types of content items includeposts referencing a specific content item (e.g., a TV show contentitem), indications of affinity (e.g., “likes” or “+1 s”) by users forother content items, and links to outside internet content (e.g., to theFOX™ website for Glee).

As an example of a content item, a first user (“User A”) may haveauthored a comment directed to another recipient user (“User B”) of theSNS, stating “Did you see the new Glee episode last night?” This examplecontent item may include a reference between the content item and thecontent item associated with the TV show Glee. Thus, both users, User Aand User B, as well as the TV show Glee, are referenced in this socialcontent item. This example content item may also include a referencebetween the content items associated with User B and with the TV showGlee, and a reference between the example content item itself, and thecontent items of both User A and User B.

References between content items may visible or invisible to users ofthe SNS. Two examples of a visible reference may be, for example, if acontent item contains a hypertext link to the content items of therecipient user and the TV show. For example, in the content item “Carl,did you see the new Glee episode last night?”, the underlined “Carl”“Glee” may each be clickable references such that when a user clicks oneither “Carl” or “Glee”, the web browser is redirected to the web pageassociated with the clicked content item. Alternatively, references maybe invisible to users. For example, if a first user posts the contentitem “Did you see the new Glee episode last night?” on a second user'sprofile page, the posted content item may contain references to thefirst user, second user, and Glee, even though the references are notvisible to the users.

A reference between a content item associated with a user and a contentitem with another object (e.g., a television show) generally indicatesan affinity by the user for the content item. For example, if a firstuser authors a content item to a second user saying “I loved lastnight's episode of Glee,” it can be inferred that both the first andsecond users have an affinity for Glee. For content items associatedwith TV shows, it is assumed that if a user is referenced in connectionwith the TV show's content item, then there is a high likelihood thatthe user has been exposed to advertisements that air during that TVshow. The system 130 uses this assumption to determine which messages tosend to which users.

The audience population engine 625 determines which populations of usersare designated to receive which targeted messages. The creation of apopulation of users may be triggered in response to a request from themessage selection engine 635, or in response to the airing of aparticular TV show or advertisement. The audience population engine 625uses both references between content items of SNS users and specific TVshows, as well as airing overlap information between those specific TVshows and the advertisements that air during those TV shows to output amapping of SNS users and advertisements. The mapping created by theaudience population engine 625 lists advertisements that SNS users whoare referenced in connection with specific TV show are likely to haveseen, and is stored in the population store 630. For a given user, thepopulation store 630 may list mappings to all advertisements they arelikely to have seen. For a given advertisement, the population store 630may list mappings to all users who are likely to have seen thatadvertisement.

In addition to mapping individual users of the SNS to advertisementsthey are likely to have seen, the audience population engine 625 is alsoconfigured to form populations of authors who may have seen a given TVshow or advertisement. The audience population engine 625 createspopulations of authors by filtering from the list of SNS users who aremapped to a given advertisement (e.g., by filtering from the list ofusers who are likely to have seen the given advertisement). Thesepopulations may be further filtered based on demographic criteria,content criteria, temporal criteria, or any number of other criteria.The users that make up each population may overlap. Populationsgenerated by the audience population engine 625 are stored in thepopulation store 630.

Demographic criteria for creating a population includes, for example,selecting SNS users based on age, gender, socioeconomic data, genreinterests, hobbies, groups, networks, affiliations, and/or location.This information may be drawn from the content items associated with orreferencing the user in the SNS and stored in the social media store260. For example, it may be specified that a particular advertisement isonly to be sent to user ages 18-29.

Content criteria for determining a population includes selecting SNSusers based on whether the content items referencing those users matchone or more content filters. Content filters may be, for example, beparticular keywords or phrases. For example, the filter may be set toinclude users who author content items that contain the name “Jane,” asin comment containing the statement: “Did you see Jane's outfit!”Filters may also be set to include users who mention particular productsor items. For example, filters may include users who use the words“outfit” or “dress” or “wearing”, such as “I love what Brenda waswearing!” Filters may also be set to include users based on theirsentiment towards products, in order to, for example, only target userswho express positive sentiment toward those products. For example, afilter may be set to include users referring to a “dress” in positivecontexts only, such as “I love that dress,” and not users who use themin negative contexts, such as “I hate that dress.”

Temporal criteria for determining a population includes selecting SNSusers who are referenced in content items created (e.g., a comment beingposted) within a certain period of time (e.g., 60 seconds) before orafter an advertisement or TV show is aired. Temporal criteria may alsobe used to remove users from a population who have recently (e.g.,within the last hour) received a message from the system 130, eitherfrom the same advertiser or from any advertiser.

Once created, populations may later be refined or generalized from theoriginal population by adjusting the criteria used to create thepopulation. The ability to tailor the populations of SNS usersfacilitates use of the system 130 by advertisers or third parties whowish to narrow or widen the scope of their targeted messagingactivities. For example, instead of targeting all users who havecommented on a particular episode of Glee, a population could bebroadened to include all users who have ever commented on any episodeGlee. The reasoning behind targeting the more general population of Gleeviewers for messaging is that users who are engaged enough to comment ona particular episode of a TV show are likely to be watching that show onfuture occasions, even if they are not referenced in a content item fora particular episode.

Message library 640 is a data store that stores messages to be sent toSNS users. Messages sent to SNS users include advertisements, brands,advertising creatives, as well as triggering messages. Triggeringmessages are messages that are sent to the SNS 110 to inform the SNSthat another message (e.g., a specific advertisement) should beforwarded to one or more SNS users. Triggering messages allow thetargeted messaging provider 130 to direct advertisements to SNS userswithout having to directly storing the advertising content. Messagelibrary 640 also stores metadata associated with each message. Messagemetadata may include advertiser, geographic or usage restrictions, andmessage sequencing information. Message library 640 also stores rulesfor when and to whom messages are to be sent. The content of a messagemay include offers, incentives, and product information. For example, amessage may take the form of a hyperlink stating “Click here for anadditional 10% off jeans already on sale at Old Navy™” along with anunderlying URL to an ecommerce site where the user can purchase thisproduct. Message library 640 receives messages to send and rules forsending them from advertisement sources 160 through data ingestion 300.

Rules for messages may take the form of a mapping between a combinationof any one of a particular advertisement or a particular TV show as ruleantecedents, and rule consequents including a particular message to besent, a particular targeted population of authors, and a particular timethe message is to be sent. For example, a rule may embody the logic of“If advertisement X airs during show Y, then send message N to thePopulation M.” As some advertisers show the same advertisement multipletimes during a TV show, the rules can also more precisely identify atime (or time window) at which an advertisement aired, the number ofmessages to be sent in response to the advertisement, or theadvertisement's sequence position (e.g., first appearance, secondappearance, etc.). Sequence position is useful where the advertiser doesnot know in advance exactly when its advertisements may appear, and toovercome variations in program scheduling.

As described above, the population of users who receive a message can bechanged by adjusting the population criteria. The population of usersreceiving a message can be changed by implementing rule antecedents orrule consequents that filter populations of users receiving a message.

Rules may also specify when a message is to be sent. For example, a rulemay specify that a message is to be sent while the advertisement isairing, within a specific time period after the advertisement airs, thenext time the recipient user logs into the SNS, the next time the userauthors a content item on the relevant TV show or advertisement, or thatthe message may be sent at anytime in the future.

Message selection engine 635 determines which messages to send to SNSusers. The sending of a message to a SNS may be triggered either on themessage selection engine's 635 own initiative, or in response to arequest by a SNS.

To send a message on its own initiative, the message selection engine635 is configured to keeps track of the TV shows and advertisements thatare currently airing or have aired. The message selection engine 635 maydo this by monitoring information from the TV show/ad overlap store 620,and/or from the TV programming guide 605. When advertisements or TVshows are detected as having aired, the message selection engine 635queries the message library 640 for rules wherein the detectedadvertisements or TV shows are used in rule antecedents.

If such a rule is matched, the message selection engine 635 identifies amessage associated with the matched rule, as well as a population of SNSwho are to be messaged according to the matched rule. If more than onerule is matched, then the message selection engine 635 may selectsbetween the possible matched rules. The selection may, for example, bebased on how recently the user is expected to have seen the ad, theamount of time since a user or population received a message, and/or howmuch an advertiser associated with a rule and message paid or bid forthe advertising space for that message,

The message selection engine 635 may also be configured to instruct thata message be sent within a particular time period (e.g., within 5minutes after a detected TV show or ad airs). The time period may bespecified by the advertiser as part of the message metadata as stored inthe message library 640. For example, an advertiser may specify thatmessage must be delivered within 30 seconds of the advertisement airing.Alternatively, the advertiser may specify that that a sequence ofmessages is be transmitted over a designated time period.

Alternatively, if a request for a message is received from the SNS, themessage selection engine 635 determines what message to send based onthe received request. The received request may specify an individualuser, or a population of users defined according to one or morecriteria.

In the case of a request for a message for a specific user, the messageselection engine 635 may query the audience population engine 625 forthe list of advertisements the user is likely to have seen. The messageselection engine 625 then queries the message library 640 for rulesassociated with the listed advertisements. If such a rule is matched,the message selection engine 635 identifies a message associated withthe matched rule to be sent to the user.

In the case of a request for a message for a population of users, themessage selection engine 635 may query the audience population engine625 for a population of users and the advertisements they are likely tohave seen based on the received criteria. The message selection engine625 then queries the message library 640 for rules associated with thelisted advertisements. If such a rule is matched, the message selectionengine 635 identifies a message associated with the matched rule to besent to the user.

A messaging interface 645 communicates messages to SNS users. Themessaging interface 645 communicates with the SNS through an API 112made available by the SNS. The messaging interface 645 is alsoconfigured to receive requests from the SNS for messages. In oneembodiment, requests received for messages by the SNS are the result ofthe system 130 or a third party (not shown) bidding for the right topresent advertisements to users or populations of users within the SNS.The bidding system may be implemented in the messaging interface 645 orin another part of the system (not shown).

FIG. 7 is a flow diagram illustrating one embodiment of a method forsocial networking system (SNS) targeted message synchronization. In thisembodiment, SNS targeted message synchronization sends messages to usersof SNSs who meet targeting criteria specified by advertisers. As above,the SNS targeted message synchronization process leverages theassumption that a SNS user who refers to a TV show or advertisement in asocial media content item is likely to have been exposed to theadvertisements that are aired during that TV show or near in time tothat advertisement. However, in this case the SNS 110, rather than thetargeted messaging provider 130, determines specifically which userswill receive a message from an advertiser. Instead, the targetedmessaging provider 130 sends the SNS 110 messages along with targetingcriteria that the SNS 110 uses to determine which users receive amessage.

The process of FIG. 7 is similar to the process described in FIG. 6 withrespect to data ingest 300, event airing detection 610, the annotatedevent store 290, the TV programming guide 605, the TV show/ad overlapengine 615, and the TV Show/Ad Overlap Store 620. One difference is thatin this case, data ingestion 300 does not receive information from thesocial networking system regarding individual users. As a consequence,in this embodiment populations of users who are likely to have seen anad are not created by the targeted messaging provider 130, and mayinstead be created by the SNS, depending upon the implementation of theSNS.

As above, message selection engine 635 determines which messages to sendto SNS users. The sending of a message to the SNS may be triggeredeither on the message selection engine's 635 own initiative, or inresponse to a request by the SNS. To send messages on its owninitiative, the message selection engine 635 may send messagesresponsive to the airing of an advertisement, responsive to a rule inthe message library 640 being matched, or according to a schedule, forexample as part of a feed to the SNS. The SNS may also request messages,either individually, in batches, or as part of a feed.

The message selection engine 635 queries the message library for rulesthat match aired, detected advertisements. If such a rule is matched,the message selection engine 635 identifies a message associated withthe matched rule, as well as one or more targeting criteria to be sentalong with the message. This is in contrast to the example of FIG. 6,where populations were instead used as a determining factor in whoreceived a message.

In one embodiment, the request for messages received from the SNS mayinclude additional rule criteria that must be matched in addition to anycriteria otherwise included in a rule. In this way, SNSs 110 can exerttheir own preferences over which messages they receive from the targetedmessaging provider 130.

The message and targeting criteria are sent to the SNS 110 by themessaging interface 645. The targeting criteria sent along with themessage informs the SNS 110 of which users are specified to receive themessage. The targeting criteria is stored in the message library 640along with messages and their associated rules. Upon receipt of amessage and targeting criteria, the SNS 110 is tasked with analyzing thetargeting criteria and determining which specific users receive themessage. The targeting criteria may, for example, include demographicinformation (e.g., age, gender), expected interests of recipient users,habits or activity of recipient users, influence of users on other SNSusers (e.g., number of friends, how often they log in to the SNS, howoften they author posts in the SNS, how recently since their last login), information regarding the television show or radio program or othertime-based media event the message is associated with, the time ofoccurrence of an airing of a time-based event associated with themessage, and other factors.

The targeting criteria may also specify the manner in which the messageis to be delivered. For example, the targeting criteria may specify thenumber of messages to be delivered to each user (e.g., one, a sequenceof three), the advertisement creative the message is associated with,and the time the message is to be delivered (e.g., within 5 minutesafter a detected ad airs). As described above, this may also includeconsiderations of how recently the user is expected to have seen the ad,the amount of time since a user or group of users have received amessage,

In one embodiment, messages may be sent from the targeted messagingprovider 130 to the SNS 110 without any targeting criteria. In thiscase, the SNS 110 chooses how to distribute messages amongst itsmembers.

Example: Old Navy™ Targeted Messaging

Old Navy™ is a retail clothing company who advertises on TV and who maywant to offer coupons to certain users. Conventionally, Old Navy™ coulduse a direct mail marketing campaign to send coupons to individualsselected from qualified mailing lists (such as prior purchasers of theirproducts), or simply allow users to download coupons from a website.Neither of these approaches directly leverage the advertisingexpenditures that the company has made for TV advertisements.

In one embodiment, Old Navy™ can send coupons as targeted messagesdirectly to SNS users who are connected, in the SNS, to TV shows inwhich Old Navy's advertisements appear. These messages may be targetedin the sense their content may be tailored based on both Old Navy'sadvertising creative campaign and also based on the TV show in which thecreative is airing. The targeted messages may include coupons offeringdiscounts to the users if they click on a URL contained in the message.

In one embodiment, one or more TV streams is monitored for airings ofspecific Old Navy™ advertisements. At some point during one of the TVstreams, an Old Navy™ commercial advertising a sale on jeans is detectedusing event airing detection 610. The TV show/ad overlap engine 615 usesthe current time and stream information (e.g., channel, media market) todetermine which TV show was being aired on that stream when theadvertisement aired. For example, it may be determined that the OldNavy™ advertisement was aired during a particular episode of the showGlee on channel 10 starting at 8 pm.

Having detected this occurrence of the Old Navy™ advertisement, and onits own initiative or responsive to a request from the SNS, the messageselection engine 635 determines if there is a rule pertaining to the OldNavy™ advertisement in the message library 640. If a rule matches theairing of the advertisement, one or more messages are sent from themessaging interface 745 to the SNS. The SNS will determine which usersreceive which message based on the rule and the targeting criteria. Themessage may also be sent to particular users of the SNS based onreferences between those users and Glee or Old Navy in social mediacontent items received from the SNS, as well based on the rule and thetargeting criteria. The message may, for example, be: “Click here for anadditional 10% off jeans already on sale at Old Navy™.”

Messages may also be more specifically targeted to the TV show in whichthe ad aired. For example, when an Old Navy™ ad airs during Glee, thefollowing message may be sent to each user in the target population:“Glee fans: click here for a Glee trivia quiz. You could win anadditional 10% off jeans already on sale at Old Navy™.” In anotherexample, when an Old Navy™ advertisement airs during the TV show Glee,the following message may be sent to users who are classified as female,age 18-24: “Click here for an additional 10% off skirts already on saleat Old Navy™.”

Additional Considerations

Although TV and advertising domains are described above, the methodsdescribed herein can be adapted to any domain using time-based media(e.g., radio). The method of adaptation is general across differentdomains. Techniques and features used for event segmentation andannotation are adapted to reflect domain specific characteristics. Forexample, detecting events in football exploits the visibility of grassas it is represented in the color distributions in a video frame, whiledetecting events in news video or audio clip may exploit clues in theclosed captioning stream.

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules orengines, without loss of generality. The described operations and theirassociated modules or engines may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modulesor engines, alone or in combination with other devices. In oneembodiment, a software module or engine is implemented with a computerprogram product comprising a computer-readable medium containingcomputer program code, which can be executed by a computer processor forperforming any or all of the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be persistently stored in a non-transitory, tangible computerreadable storage medium, or any type of media suitable for storingelectronic instructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

1. A computer-executed method for sending messages to users of socialnetworking systems, comprising: accessing a content item from a socialnetworking system, the content item comprising references to a user ofthe social networking system, and to at least one selected from a groupconsisting of a television media event and an advertisement media event;accessing a video stream broadcasted by an external broadcasting entity,the video stream comprising the television media event and theadvertisement media event; analyzing, using a computer processor, thevideo stream to determine that the advertisement media event has airedduring an airing of the television media event; sending the user amessage through the social networking system, the sending of the messagebased on the determination and the references.
 2. The computer-executedmethod of claim 1, wherein accessing the video stream comprising theadvertisement media event comprises: accessing a plurality ofadvertisement metadata instances; segmenting the video stream intosegments corresponding to the media events, each segment having abeginning and an end; and determining, for each advertisement metadatainstance, a segment of the video stream that most likely aligns with theadvertisement metadata instance.
 3. The computer-executed method ofclaim 1, wherein analyzing the video stream to determine that theadvertisement media event has aired during the airing of the televisionmedia event comprises: extracting event features from metadataannotations associated with the media events; and mapping the eventfeatures of the advertisement media event to the event features of thetelevision media event, the event features including an airing time anddate.
 4. The computer-executed method of claim 1, further comprising:determining that at least one selected from the group consisting of thetelevision media event and the advertisement media event has airedwithin a recent time period; determining whether the user matches a ruleindicating when the message is to be sent to the user through the socialnetworking system; and responsive to the airing and the user matchingthe rule, sending the message.
 5. The computer-executed method of claim1, further comprising: receiving a request for an advertisement from thesocial networking system; determining whether the user matches a ruleindicating conditions for sending the message; and responsive to therequest and the user matching the rule, sending the message.
 6. Thecomputer-executed method of claim 1, further comprising creating amapping between the advertisement media event and the user based on thereferences and the determination that the advertisement media event hasaired during the airing of the television media event, and wherein themessage is sent based on the mapping.
 7. The computer-executed method ofclaim 1, wherein a message content of the message is based on at leastone selected from a group consisting of the references and thedetermination that the advertisement media event has aired during anairing of the television media event.
 8. A computer-executed method forsending messages to users of social networking systems, comprising:accessing a plurality of content items from a social networking system,each content item referencing at least one user of the social networkingsystem and referencing at least one selected from a group consisting ofa television media event and an advertisement media event; accessing avideo stream broadcasted by an external broadcasting entity, the videostream comprising the television media event and the advertisement mediaevent; analyzing the video stream to determine that the advertisementmedia event has aired during an airing of the television media event;creating a plurality of mappings indicating users of the socialnetworking system referenced by content items that also reference atleast one selected from the group consisting of the television mediaevent and the advertisement media event; forming a population of usersbased on the mappings; and sending a message to the population of usersthrough the social networking system, the sending of the message basedon the mappings.
 9. The computer-executed method of claim 8, wherein thepopulation includes only those users meeting at least one selected froma group consisting of a demographic criterion, a content criterion, anda time criterion.
 10. The computer-executed method of claim 8, furthercomprising: determining that the television media event has aired withina recent time period; and responsive to the airing, sending the message.11. The computer-executed method of claim 8, further comprising:receiving a request for an advertisement from the social networkingsystem; and responsive to the request, sending the message.
 12. Thecomputer-executed method of claim 8, further comprising: determiningwhether the population matches a rule indicating conditions for sendingthe message; and responsive to the rule being matched, sending themessage.
 13. The computer-executed method of claim 8, furthercomprising: determining whether the population matches a rule indicatingconditions for sending the message; and responsive to the rule beingmatched, sending the message.
 14. (canceled)
 15. The computer-executedmethod of claim 8, further comprising creating a mapping between theadvertisement media event and the user based on the references and thedetermination that the advertisement media event has aired during theairing of the television media event, and wherein the message is sentbased on the mapping.
 16. A computer-executed method for sendingmessages to users of social networking systems, comprising: accessing avideo stream comprising a television media event and an advertisementmedia event; accessing a video stream broadcasted by an externalbroadcasting entity, the video stream comprising the television mediaevent and the advertisement media event; analyzing the video stream todetermine that the advertisement media event has aired during an airingof the television media event; determining whether the rule is matchedbased upon the airing of the advertisement media event; and responsiveto the rule being matched, sending the message to the social networkingsystem.
 17. The computer-executed method of claim 16, furthercomprising: receiving a request from the social networking system;responsive to the request, sending the message to the social networkingsystem.
 18. The computer-executed method of claim 16, wherein themessage is sent as part of a feed to the social networking system. 19.The computer-executed method of claim 16, wherein the message is sent ina batch to the social networking system along with a plurality of othermessages.
 20. The computer-executed method of claim 16, furthercomprising: accessing a targeting criterion indicating conditions forreceiving the advertisement; responsive to the rule being matched,sending the targeting criterion with the message to the socialnetworking system.
 21. The computer-executed method of claim 20, whereinthe targeting criterion includes at least one selected from a groupconsisting of: demographic information, interests, television mediaevent information, advertisement media event information, and time ofoccurrence of the airing.
 22. The computer-executed method of claim 1,wherein accessing the content item from the social networking systemcomprises determining a confidence score for the content item indicativeof the probability that the content item is relevant to the televisionmedia event.
 23. The computer-executed method of claim 22, whereindetermining the confidence score comprises: extracting event featuresfrom annotations associated with the television media event; extractingsocial media features from the content item; and identifying arelationship between the event features and social media features, theconfidence score of the candidate social media content item being basedat least partially on the relationship.